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Method/System for Estimating the Orientation of a Segment of Digital Ink 



Technical Field 

5 [001] The present invention broadly relates to pen-based computing systems and 
handwriting (digital ink) recognition systems, and in particular, to a method of 
estimating the orientation of a segment of digital ink generated using a pen-based 
computing system, and to a pen-based computing system for estimating the 
orientation of a segment of digital ink. The estimated orientation of the segment of 

10 digital ink can then be subsequently used in a digital ink line orientation 
normalization technique. 
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Background Art 

[003] Digital ink processing systems must deal with the huge variability in 
handwriting and drawing that occurs due to the differing styles of individual 
10 writers. As a result, most systems perform a number of pre-processing steps to 
limit this variation. Examples of such systems include handwriting recognition 
systems, digital signature verification systems, document analysis systems, and 
digital ink searching systems. 

15 [004] An instance of such a procedure is orientation normalization which is used 
to reduce the variance of the input by aligning the digital ink as if it was written 
using a standard orientation on the page (for example, written left-to-right on a 
horizontal line for Latin character based scripts). By aligning the digital ink in 
such away, the ink processing system can ignore the effects of variation in 

20 orientation, and as such can be made simpler, more robust, and more accurate. 

[005] Orientation normalization is usually performed as one of the first steps in a 
digital ink processing system, and is used to minimize error in later stages (for 
example, line, word, and character segmentation, feature extraction, etc.) 
25 Generally, the angle of a segment of digital ink relative to a standard reference 
angle (e.g. horizontal) is estimated and used to re-orient the digital ink such that 
the angle of digital ink matches the reference angle. 

[006] Orientation normalization for Latin character scripts is often performed 
30 using baseline correction; where the baseline of a line of text is defined as the 
imaginary natural line on which a user places characters that do not have 
descenders (e.g. "a", "b", "c", "d", "e", "f», »h", etc.). This is done by estimating 



WO 2004/042644 



4 



PCT/AU2003/001342 



the baseline of a segment of digital ink and then rotating the ink to be horizontal. 
Whilst most systems assume baselines are roughly linear, some systems attempt to 
model baseline drift using more sophisticated models such as splines [A. Hennig, 
N. Sherkat, and R. Whitrow, "Zone Estimation for Multiple Lines of Handwriting 
5 using Approximating Spline Functions", Fifth International. Workshop on 
Frontiers in Handwriting Recognition (IWFHR), pp. 325-328, September 1996]. 

[007] A significant amount of research has been performed on orientation 
estimation and normalization for digital ink, with particular emphasis on 

10 techniques that are applicable to Optical Character Recognition systems. Early 
research systems relied on heuristics and empirical thresholds [W. Guerfali and R. 
Plamondon, "Normalization and restoring on-line handwriting", Pattern 
Recognition, 26 (3), pp. 419-431, 1993; S. Madhvanath and V. Govindaraju, 
"Using holistic features in handwritten word recognition", United States Postal 

15 Services (USPS), pp. 183-198, 1992], along with simple techniques such as linear 
regression through stroke minima [R. Bozinocic and S. Srihari, "Off-line cursive 
script word recognition"; IEEE Transactions of Pattern Analysis and Machine 
Intelligence 11, pp. 69-83, 1989]. Due to the brittle nature of these techniques, 
more sophisticated systems using projection profiles [A. Vinciarelli and J. Luettin, 

20 "A New Normalization Technique for Cursive Handwritten Words", Pattern 
Recognition Letters 22, pp. 1043-1050, 2001; M. Brown and S. Ganapathy, 
"Preprocessing Techniques for Cursive Script Word 1 Recognition", Pattern 
Recognition 16 (5), pp. 447-458, 1983] and generalized projections [G. Nicchiotti 
and C. Scagliola, "Generalised Projections: a Tool for Cursive Handwriting 

25 Normalisation", Fifth International Conference on Document Analysis and 
Recognition (ICDAR), September 1999] were developed. Other techniques have 
sis.se been developed, including: least squares and weighted least-squares [M. 
Morita, S. Games, J. Facon, F. Bortolozzi, and R. Sabourin, "Mathematical 
Morphology and Weighted Least Squares to Correct Handwriting Baseline Skew", 

30 Fifth International Conference on Document Analysis and Recognition (ICDAR), 
pp. 430-433, September 1999; T. Breuel, "Robust least square baseline fording 
using a branch and bound algorithm", Proceedings of the SPIE, pp. 20-27, 2002], 
geometric modelling and pseudo-convex hull [M. Morita, F. Bortolozzi, J. Facon, 
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and R. Sabourin, "Morphological approach of handwritten word skew correction", 
SIBGRAPI f 98, International Symposium on Computer Graphics, Image Processing 
and Vision, Rio de Janeiro, Brazil, pp- 456-461, October 1998], techniques based 
on the Hough transform [A. Rosenthal, J. Hu and M. Brown, "Size and orientation 
5 normalization of on-line handwriting using Hough transform", ICASSP f 97, 
Munich, Germany, April 1997], model based methods [Y. Bengio and Y. LeCun, 
"Word normalization for on-line handwritten word recognition", Proceedings of the 
International Conference on Pattern Recognition, pp. 409-413, October 1994], 
skew detection using Principal Component Analysis [Steinherz, N., Intrator, and E. 

10 Rivlin. "Skew detection via principal components analysis", Proceedings of the 
International Conference on Document Analysis and Recognition (ICDAR), pp. 
153-156,1999], and baseline estimation using approximating spline functions [A. 
Hennig, N. Sherkat, and R. Whitrow, "Zone Estimation for Multiple Lines of 
Handwriting using Approximating Spline Functions", Fifth International. 

15 Workshop on Frontiers in Handwriting Recognition (IWFHR), pp. 325-328, 
September 1996]. 

[008] Some orientation normalization techniques have been disclosed in prior art 
patent specifications, including the use of boundary projections combined with the 

20 Hough transform [T. Syeda-Mahmood, "Method of grouping handwritten word 
segments in handwritten document images", US Patent 6,108,444]; a system for 
digit normalization of scanned images that works by finding the bounds of a 
parallelogram that completely encloses the character image [R. Vogt, "Handwritten 
digit normalization method", US Patent 5,325,447;3]; methods that use linear 

25 projection and a clustering algorithm to detect elements in a histogram that 
correspond to ascender, descender, and base lines [W. Bruce, et al, "Estimation of 
baseline, line spacing and character height: for handwriting recognition", US 
Patent 5,396,566; J. Kim, "Baseline Drift Correction of Handwritten Text", IBM 
Technical Disclosure Bulletin 25 (10), March 1983]; and a least squares 

30 calculation combined with rotation around a centroid for the normalization of 
signatures [F. Sinden and G. Wilfong, "Method of normalizing handwritten 
symbols", US Patent 5,537,489] in an online signature verification system. 
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[009] Whilst the techniques described above are sometimes effective, they suffer 
from a number of significant limitations. For example, many assume that all lines 
of written text are oriented at the same angle on the page, and thus cannot handle 
pages of arbitrarily rotated text lines. Other limitations include the fact that the 
5 algorithms require significant processing resources (e.g. Hough transform), are 
quantised (e.g. Hough transform), do not work well for short segments of text (e.g. 
projection methods), are brittle due to empirically estimated thresholds (heuristic 
and rule-based techniques), or are sensitive to ascenders, descenders and outliers 
(e.g. least squares regression and projection techniques). 

10 

[010] The azimuth of a writing implement is defined in [R. Poyner, "Wintab 
Interface Specification 1.1: 16- and 32-bit API Reference" LCS/Telegraphics] as 
the "clockwise rotation of the cursor about the z axis through a full circular range". 
In other words, if x and y define the horizontal and vertical axes of a sheet of 
15 paper, and z defines the axis that is normal to the paper, the azimuth is the rotation 
of the pen about the z axis. Some pen-based computing systems are able to 
measure the azimuth of a writing implement during the generation of digital ink, 
including Wacom graphics tablets and Netpage pens [K. Silverbrook et al, 
"Sensing Device", WO 02/42989]. 

20 

Disclosure Of Invention 

[011] In the preferred embodiment, the invention is configured to work with the 
Netpage networked computer system, a detailed description of which is given in 
our co-pending applications, including in particular PCT application WO0242989 

25 entitled "Sensing Device" filed 30 May 2002, PCT application WO0242894 
entitled "Interactive Printer" filed 30 May 2002, PCT application WO02 14075 
"Interface Surface Printer Using Invisible Ink" filed 21 February 2002, PCT 
application WO0242950 "Apparatus For Interaction With A Network Computer 
System" filed 30 May 2002, and PCT application WO03034276 entitled "Digital 

30 Ink Database Searching Using Handwriting Feature Synthesis" filed 24 April 2003. 
It will be appreciated that not every implementation will necessarily embody all or 
even most of the specific details and extensions described in these applications in 
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relation to the basic system. However, the system is described in its most complete 
form to assist in understanding the context in which the preferred embodiments and 
aspects of the present invention operate. 

[012] In brief summary, the preferred form of the Netpage system provides an 
interactive paper-based interface to online information by utilizing pages of 
invisibly coded paper and an optically imaging pen. Each page generated by the 
Netpage system is uniquely identified and stored on a network server, and all user 
interaction with the paper using the Netpage pen is captured, interpreted, and 
stored. Digital printing technology facilitates the on-demand printing of Netpage 
documents, allowing interactive applications to be developed. The Netpage printer, 
pen, and network infrastructure provide a paper-based alternative to traditional 
screen-based applications and online publishing services, and supports user- 
interface functionality such as hypertext navigation and form input. 

[013] Typically, a printer receives a document from a publisher or application 
provider via a broadband connection, which is printed with an invisible pattern of 
infrared tags that each encodes the location of the tag on the page and a unique 
page identifier. As a user writes on the page, the imaging pen decodes these tags 
and converts the motion of the pen into digital ink. The digital ink is transmitted 
over a wireless channel to a relay base station, and then sent to the network for 
processing and storage. The system uses a stored description of the page to 
interpret the digital ink, and performs the requested actions by interacting with an 
application. 

[014] Applications provide content to the user by publishing documents, and 
process the digital ink interactions submitted by the user. Typically, an application 
generates one or more interactive pages in response to user input, which are 
transmitted to the network to be stored, rendered, and finally printed as output to 
the user. The Netpage system allows sophisticated applications to be developed by 
providing services for document publishing, rendering, and delivery, authenticated 
transactions and secure payments, handwriting recognition and digital ink 
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searching, and user validation using biometric techniques such as signature 
verification. 



[015] Generally, the present invention seeks to provide a method for estimating 
the orientation of a segment of digital ink using pen orientation information. In one 
form, the technique involves using training data to build a pen orientation model, 
which can be for an individual writer, which is used to estimate the orientation of 
subsequently written digital ink. 



[016] The digital ink orientation estimation technique described herein seeks to 
overcome or ameliorate the limitations described in the prior art, and improve on 
current techniques by utilizing pen orientation information that has been 
previously unavailable or ignored by other systems. Additionally, one form of the 
invention can use training data to generate a writer-dependent pen orientation 
model that is used during orientation estimation. 



[017] In a broad form the present invention provides a method of estimating the 
orientation of a segment of digital ink, the method including the steps of: 
measuring the azimuth of the pen at a sampling rate during writer generation of the 
segment of digital ink; and estimating the orientation of the segment of digital ink 
using the measured azimuth of the pen at sampled points. 

[018] Preferably, the estimated orientation of the segment of digital ink is 
subsequently used in a digital ink line orientation normalization technique. In 
accordance with specific embodiments, a single, fixed orientation estimation is utilised 
for a line of digital ink, or, an orientation estimation that varies across a line of digital ink is 
utilised. 



[019] According to a further possible form of the invention, the orientation of the 
pen at sampled points is estimated by subtracting the mean azimuth of a digital ink 
training sample from the measured azimuth of the sampled points, and normalizing 
the estimated orientation to be within the range of 0° to 360°. 



WO 2004/042644 



9 



PCT/AU2003/001342 



[020] According to other specific embodiments, the segment of digital ink is more 
than one character of digital ink. Also, the segment of digital ink may be a line 
segment. In this form, line segmentation may be performed by measuring a change 
in azimuth value. 

[021] In yet further specific embodiments of the present invention, the orientation 
estimation uses a writer independent handwriting model; the orientation estimation 
uses a writer dependent handwriting model trained using sample digital ink input 
by the writer; or the writer dependent handwriting model is trained using sample 
digital ink input by the writer using a consistent baseline. 

[022] In a further broad form the present invention provides a pen-based 
computing system for estimating the orientation of a segment of digital ink, the 
system including a pen-based computing pen to input digital ink, and a processor 
15 adapted to estimate the orientation of a segment of digital ink by measuring the 
azimuth of the pen at a sampling rate during writer generation of the segment of 
digital ink, and estimating the orientation of the segment of digital ink using the 
measured azimuth of the pen at sampled points. 

20 [023] In still yet a further broad form the present invention provides a pen-based 
computing system for estimating the orientation of a segment of digital ink, the 
system including: 

(1) a pen-based computing pen to input digital ink; 

(2) a storage unit to store a handwriting model; 
25 (3) a processor, the processor being adapted to: 

(a) retrieve the handwriting model; 

(b) receive a measurement of the azimuth of the pen at a sampling 
rate during writer generation of the segment of digital ink; and 

(c) estimate the orientation of the segment of digital ink by 
30 modifying the measured azimuth of the pen at sampled points 

using the handwriting model. 
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Brief Description of Figures 

[024] The present invention should become apparent from the following 
description, which is given by way of example only, of a preferred but non- 
limiting embodiment thereof, described in connection with the accompanying 
5 figures. 

[025] Figure 1 illustrates the average (bold), minimum, maximum, and standard 
deviation (dashed) azimuth measurements for left-handed and right-handed writers; 

10 [026] Figure 2 illustrates the training and normalization procedure; 

[027] Figure ^ illustrates measured azimuth vectors of the pen at a sampling rate 
during writer generation of the segment of digital ink; and 

15 [028] Fjgure_4 illustrates the estimated orientation of the segment of digital ink 
shown in Figure 3 after the handwriting model is applied. 

[029] |ig ure 5 illustrates a pen-based computing system. 

20 Modes For Carrying Out The Invention 

[030] The following modes are described as applied to the description and claims 
in order to provide a more precise understanding of the subject matter of the 
present invention. 

25 Azimuth Measurements 

[031] A digitising tablet was used to measure the azimuth of a pen during the 
generation of handwriting by five different writers. Digital ink was collected using 
a Wacom Intuos graphics tablet with a sampling rate of 100 Hz. The data 
collection application was developed using the Wintab Programmer Development 

30 Kit Version 1.26 [R. Poyner, "Wintab Interface Specification 1.1: 16- and 32-bit 
API Reference", LCS/Telegraphics, May 9, 1996]. 
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[032] Table 1 details the azimuth measurements for the sample data collected, 
where the angles are measured clockwise with 0° representing a vertical line 
pointing to the top of the page. Note that the azimuth measurements reveal that 
writer 2 is left-handed. Table 2 details the average, minimum, maximum, and 
standard deviation of the azimuth measurements for the sample data for both left- 
and right-handed writers, with this data illustrated in Figure 1. 



Table 1. Average, minimum and maximum azimuth measurements 





Handedness 


Average 


Minimum 


Maximum 


Writer 1 


Right 


130° 


109° 


148° 


Writer 2 


Left 


294° 


275° 


316° 


Writer 3 


Right 


149° 


124° 


173° 


Writer 4 


Right 


141° 


123° 


157° 


Writer 5 


Right 


133° 


119° 


159° 



Table 2. Average, standard deviation, minimum and maximum measurements 



Handedness 


Average 


Std. Deviation 


Minimum 


Maximum 


Right 


138° 


11.9° 


109° 


173° 


Left 


294° 


8.2° 


275° 


316° 



[033] As can be noted from the results the azimuth of a pen during handwriting is 
relatively stable for a particular writer (as can be seen by the small standard 
deviation and difference in the minimum and maximum values). 



Orientation Estimation 

[034] To estimate the orientation of digital ink using the azimuth measurements, a 
handwriting model is required to be available. Whilst the technique works with a 
small number of writer-independent models (e.g. one for left-handed writers and 
another for right-handed writers) that do not require training, more accurate results 
are achieved using a writer-dependent model that is trained using sample input 
from the writer. To do this, the system is trained using digital ink data that was 
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written using a consistent, well-defined baseline. This data can be derived from 
normal input (for example, form input data that is constrained to be written 
horizontally) or from a separate training procedure. The training data is then used 
to generate a model for the writer as shown in Figure 2. 

[035] Alternatively, training can occur using arbitrary handwritten input (i.e. 
without explicit training data) by using an alternative orientation estimation 
technique to truth the data from which the writer-dependent model is generated. 
Since the training data does not need to be large (a few letters can be sufficient), 
the technique used to truth the data can be very expensive (processor intensive) 
since it is only run once on a small segment of ink. In addition to this, algorithms 
that fail in some situations can be used, since a single successful orientation 
estimation is all that is required for the training procedure. 

[036] Once the model has been generated, it can be used for line segmentation and 
orientation estimation and normalization. When performing line segmentation, a 
large jump in the azimuth value (e.g. a value larger than the expected variance as 
given by the writer-specific model) is an indication of the start of a new line of 
text with an orientation different from that of the previous line. For orientation 
normalization, the model can be used to generate an estimate of the text orientation 
for the line segment, which is then used to perform baseline normalization. 

[037] As an example, Figure 3 depicts the measured azimuth vectors for a stream 
of digital ink handwriting of "hello world". The vectors represent the orientation of 
the pen during the generation of the ink, and the angle of the pen relative to the 
page can be seen to change smoothly during the writing. To estimate the 
orientation of the digital ink in this example, a simple mean azimuth model was 
used, where the mean of the azimuth values in a set of training data (a single 
horizontal line of text written by the same writer) was calculated and stored: 



n 



a = 
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where a { is the azimuth measurement in degrees at sample i, and n is the number of 
samples in the digital ink. 

[038] The mean value represents the normal azimuth that the writer holds the pen 
relative to the page when writing. To estimate the orientation at each sample point, 
the mean values were subtracted from the azimuth values of the digital ink 
example shown in Figure 3, with the results normalized to ensure they are in the 
range 0° - 360°: 

r a, -a +360° a,-a<0° 
a i - \ _ (2) 

[ a, — a otherwise 

where a\ is the azimuth measurement in degrees at sample i, and a is the mean 
azimuth value for the writer (as calculated previously). 

[039] The mean azimuth value derived from the training data, was approximately 
130°. This value was subtracted from each of the measured azimuth values shown 
in Figure 3 giving the normalized orientation estimates are shown in Figure 4. As 
can be seen in Figure 4, the estimated orientation vectors give a good indication of 
the curved baseline orientation for the text "hello world" at each point on the 
digital ink path. These orientation vectors can then be used for digital ink line 
orientation normalization. 

[040] Once the ink baseline orientation has been estimated, a number of digital 
ink line orientation normalization techniques are possible. A simple method for 
text written with a linear baseline is to find the mean estimated orientation of the 
samples in the digital ink segment, and rotate the ink to counter this orientation. 
More sophisticated techniques include using a smoothed running estimate of the 
orientation, or fitting a curve (e.g. spline) to the estimated orientation vectors and 
warping the digital ink segment to ensure the estimated baseline is horizontal and 
linear. 
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[041] This demonstrates a specific, but non-limiting, example method for 
estimating the orientation of digital ink using the azimuth of the writing device and 
the use of this measurement to perform orientation normalization line segmentation 

5 [042] A further particular embodiment of the present invention can be realised 
using a pen-based processing system, an example of which is shown in figure 5. 
The processing system 10 generally includes at least a processor 11, a memory 12, 
a pen-based input device 13 and an output device 14, coupled together via a bus 
15. An external interface 16 can also be provided for coupling the processing 

10 system 10 to a storage device 17 which houses a database 18. The memory 12 can 
be any form of memory device, for example, volatile or non- volatile memory, solid 
state storage devices, magnetic devices, etc. The output device 14 can include, for 
example, a display device, monitor, printer, etc. The storage device 17 can be any 
form of storage means, for example, volatile or non-volatile memory, solid state 

15 storage devices, magnetic devices, etc. 

[043] In use, the processing system 10 is adapted to allow data or information to 
be stored in and/or retrieved from the memory 12 and/or the database 17. The 
processor 11 receives instructions via the input device 13 and can display results to 
20 a user (or writer using the pen input device 13) via the output device 14. It should 
be appreciated that the processing system 10 may be any form of processing 
system, computer terminal, server, specialised hardware, or the like. 

[044] Thus, there has been provided in accordance with the present invention, a 
25 method and system for estimating the orientation of a segment of digital ink 
generated using a pen-based computing system. 

[045] The invention may also be said to broadly consist in the parts, elements and 
features referred to or indicated herein, individually or collectively, in any or all 
30 combinations of two or more of the parts, elements or features, and where specific 
integers are mentioned herein which have known equivalents in the art to which 
the invention relates, such known equivalents are deemed to be incorporated herein 
as if individually set forth. 
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[046] Although the preferred embodiment has been described in detail, it should 
be understood that various changes, substitutions, and alterations can be made 
herein by one of ordinary skill in the art without departing from the scope of the 
5 present invention. 



